Supplementary Material: Online Kernel Learning with a Near Optimal Sparsity Bound

A. Proof of Theorem 2 We here prove a lower bound on the number of support vectors to achieve the optimal regret bound. First, we construct a set of n examples T1 = {(xi, yi)}i=1, where 〈κ(xi, ·), κ(xj , ·)〉Hκ = δij and yi ∈ {1,−1}. To make the construction, consider the degree-d polynomial kernel κ(x,y) = (xy) and an Euclidean space R where m > n. Since m > n, we can find a set of orthonormal vectors {x1, . . . ,xn} in R such that xi xj = 0 when i 6= j and xi xi = 1. It is easy to verify that this construction satisfies our assumption κ(xi,xj) = δij . For the Gaussian kernel, when the distance between xi and xj are large enough, we also have κ(xi,xj) ≈ δij . Based on T1, we construct another set T2: (z, u) ∈ T2 if there exist an index j ∈ [n] and a function ξ ∈ Hκ such that κ(z, ·) = κ(xj , ·) + ξ, u = yj , and 〈ξ, κ(xi, ·)〉Hκ = 0, ∀i ∈ [n]. (13) Thus, for each (z, u) ∈ T2, there is a corresponding (xj , yj) ∈ T1 such that the relationships in (13) hold. The existence of T2 can be proved in a similar way as that of T1. Second, we select T distinct training examples (z1, u1), . . . , (zT , uT ) from T2 such that, for each (x, y) ∈ T1 there are T/n examples constructed from it. Taking logit loss l(y, z) = ln(1 + exp(−yz)) as an example. From the above constructions, it is easy to check that